{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some features to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  10.014059116977283\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.749784\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.977917658761159\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.987811912159426\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.30074\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  5.44560814873387e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specifically, if the constructor of the net receives a value that is not 1 for the `dropout` parameter, then the net should add dropout immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.3004790897684924\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.302371489704412\n",
      "W1 relative error: 1.90e-07\n",
      "W2 relative error: 4.76e-06\n",
      "W3 relative error: 2.60e-08\n",
      "b1 relative error: 4.73e-09\n",
      "b2 relative error: 1.82e-09\n",
      "b3 relative error: 1.70e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.3042759220785896\n",
      "W1 relative error: 3.11e-07\n",
      "W2 relative error: 1.84e-08\n",
      "W3 relative error: 5.35e-08\n",
      "b1 relative error: 2.58e-08\n",
      "b2 relative error: 2.99e-09\n",
      "b3 relative error: 1.13e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "  \n",
    "  # Relative errors should be around e-6 or less; Note that it's fine\n",
    "  # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856643\n",
      "(Epoch 0 / 25) train acc: 0.260000; val_acc: 0.184000\n",
      "(Epoch 1 / 25) train acc: 0.416000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.482000; val_acc: 0.276000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.277000\n",
      "(Epoch 4 / 25) train acc: 0.600000; val_acc: 0.271000\n",
      "(Epoch 5 / 25) train acc: 0.708000; val_acc: 0.299000\n",
      "(Epoch 6 / 25) train acc: 0.722000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.832000; val_acc: 0.255000\n",
      "(Epoch 8 / 25) train acc: 0.878000; val_acc: 0.269000\n",
      "(Epoch 9 / 25) train acc: 0.902000; val_acc: 0.275000\n",
      "(Epoch 10 / 25) train acc: 0.890000; val_acc: 0.261000\n",
      "(Epoch 11 / 25) train acc: 0.930000; val_acc: 0.279000\n",
      "(Epoch 12 / 25) train acc: 0.962000; val_acc: 0.301000\n",
      "(Epoch 13 / 25) train acc: 0.964000; val_acc: 0.305000\n",
      "(Epoch 14 / 25) train acc: 0.964000; val_acc: 0.312000\n",
      "(Epoch 15 / 25) train acc: 0.966000; val_acc: 0.307000\n",
      "(Epoch 16 / 25) train acc: 0.986000; val_acc: 0.305000\n",
      "(Epoch 17 / 25) train acc: 0.986000; val_acc: 0.312000\n",
      "(Epoch 18 / 25) train acc: 0.996000; val_acc: 0.327000\n",
      "(Epoch 19 / 25) train acc: 0.990000; val_acc: 0.323000\n",
      "(Epoch 20 / 25) train acc: 0.998000; val_acc: 0.314000\n",
      "(Iteration 101 / 125) loss: 0.000145\n",
      "(Epoch 21 / 25) train acc: 0.998000; val_acc: 0.307000\n",
      "(Epoch 22 / 25) train acc: 0.996000; val_acc: 0.303000\n",
      "(Epoch 23 / 25) train acc: 0.976000; val_acc: 0.301000\n",
      "(Epoch 24 / 25) train acc: 0.980000; val_acc: 0.290000\n",
      "(Epoch 25 / 25) train acc: 0.984000; val_acc: 0.304000\n",
      "0.25\n",
      "(Iteration 1 / 125) loss: 17.318479\n",
      "(Epoch 0 / 25) train acc: 0.230000; val_acc: 0.177000\n",
      "(Epoch 1 / 25) train acc: 0.378000; val_acc: 0.243000\n",
      "(Epoch 2 / 25) train acc: 0.402000; val_acc: 0.254000\n",
      "(Epoch 3 / 25) train acc: 0.502000; val_acc: 0.276000\n",
      "(Epoch 4 / 25) train acc: 0.528000; val_acc: 0.298000\n",
      "(Epoch 5 / 25) train acc: 0.562000; val_acc: 0.297000\n",
      "(Epoch 6 / 25) train acc: 0.628000; val_acc: 0.291000\n",
      "(Epoch 7 / 25) train acc: 0.622000; val_acc: 0.299000\n",
      "(Epoch 8 / 25) train acc: 0.684000; val_acc: 0.312000\n",
      "(Epoch 9 / 25) train acc: 0.716000; val_acc: 0.289000\n",
      "(Epoch 10 / 25) train acc: 0.724000; val_acc: 0.297000\n",
      "(Epoch 11 / 25) train acc: 0.760000; val_acc: 0.309000\n",
      "(Epoch 12 / 25) train acc: 0.788000; val_acc: 0.285000\n",
      "(Epoch 13 / 25) train acc: 0.822000; val_acc: 0.315000\n",
      "(Epoch 14 / 25) train acc: 0.826000; val_acc: 0.349000\n",
      "(Epoch 15 / 25) train acc: 0.850000; val_acc: 0.340000\n",
      "(Epoch 16 / 25) train acc: 0.860000; val_acc: 0.305000\n",
      "(Epoch 17 / 25) train acc: 0.848000; val_acc: 0.300000\n",
      "(Epoch 18 / 25) train acc: 0.876000; val_acc: 0.325000\n",
      "(Epoch 19 / 25) train acc: 0.882000; val_acc: 0.340000\n",
      "(Epoch 20 / 25) train acc: 0.872000; val_acc: 0.303000\n",
      "(Iteration 101 / 125) loss: 4.189651\n",
      "(Epoch 21 / 25) train acc: 0.912000; val_acc: 0.322000\n",
      "(Epoch 22 / 25) train acc: 0.896000; val_acc: 0.309000\n",
      "(Epoch 23 / 25) train acc: 0.902000; val_acc: 0.308000\n",
      "(Epoch 24 / 25) train acc: 0.912000; val_acc: 0.329000\n",
      "(Epoch 25 / 25) train acc: 0.904000; val_acc: 0.309000\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.25]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4gAAAJNCAYAAABz4lt9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XuYXWV9//331zDIiJhwiEgygQRJg5gEYgcQoYCkEFQOATWCB8RWEQulTw+poU+bJ+ZXCyVUfChUGysCHoCUYgwKRAVR8YBMTJwIGImIZCZYwyERdYAQvr8/9p6wM8wkmWT2Xnv2fr+ua649615rr/XdszdDPnPf674jM5EkSZIk6WVFFyBJkiRJqg8GREmSJEkSYECUJEmSJJUZECVJkiRJgAFRkiRJklRmQJQkSZIkAQZESZKIiBER8buI2L/oWiRJKlK4DqIkabiJiN9VbL4CeBbYVN7+cGZ+sfZVSZI0/BkQJUnDWkQ8AnwwM7+5lWN2yczna1dV7TTya5Mk1Z5DTCVJDSci/jkiboqIGyLiaeC9EXFURPwwItZHxGMRcWVEtJSP3yUiMiLGl7e/UN5/e0Q8HRE/iIgJA1zrZRFxc0T8unzuuyPidRX7XxERV0TEoxGxISK+ExEvL+87tlzThohYExHvK7ffExHnVpzjgxFxd59a/yIiVgM/K7dfFRFdEfHbiLgvIt5U8fxdIuKfIuIX5f0dETEmIv4zIv61z+u5PSIu3Pl3QZI0HBkQJUmN6gzgS8BI4CbgeeCvgH2Ao4GTgQ9v5fnvBv4J2At4FPg/Wzn2q8BE4DXAT4HPV+y7ApgKHFk+1z8AL5QD59eATwB7A9OAlYN4facBhwNTytv3lq+zF3Az8N+9QRSYDbyD0mseBXwQeAa4Dnh3RLwMICL2BY4DbhxEHZKkBmJAlCQ1qnsy89bMfCEzezLzvsy8NzOfz8yHgYWUwtBAbs7MjszcCHwROKy/g8rnvzYzn87MZ4B5wB9HxO4RMQI4F7goMx/LzE2ZeU/5nO8F7sjMReWaHs/MFYN4ff+SmU9lZk+5js9n5pPl4aaXAa8CDiof+0HgHzLzoXK9K8rHfh/oqfg5nA18MzMfH0QdkqQGYkCUJDWqNZUbEXFwRHytPBT0t8B8Sr2JA/l1xfd/AF7Z30HlGVAvi4iHy+ddXd61D7AvsCvwi36eOm6A9u3V9/X9fUT8LCI2AE8Bu/Pi69vata6nFFYpP35+gOMkSU3AgChJalR9Z2H7T0rDPw/KzFcBc4EYguucA7wVOIHScNbeXrsA/hd4DnhtP89bM0A7wO8pzc7a6zX9HLP59UXEm4G/Ad5OaQjpnsDvePH1be1anwfOjIhp5WNuHeA4SVITMCBKkprFHsAG4PflSWS2dv/hYM/7LPAEpVD38d4dmbkJuBb4ZES8ptzbeHR5cpwvACdHxNvLk8jsExGHlp+6Anh7RLRGxB8Bf7YdNTwPPA60UBrmunvF/v8C/jkiXhslh0XEXuUaf1W+3nXAf5eHyUqSmpQBUZLULP4WeD/wNKXexJuG6LyfA9aWv+4Hvt9n/18DDwLLgCeBf6G0zNQvgVOBj5bbf8yLE85cTqmH8DfANZTC5NbcBnwTeAh4BPgt8FjF/gXAYuDO8r6FwG4V+68rX9vhpZLU5FwHUZKkJhcRJwCfBQ5M/2EgSU3NHkRJkppYROxKafmPzxgOJUkGREmSmlRETKE04+lewJUFlyNJqgMOMZUkSZIkAfYgSpIkSZLKDIiSJEmSJAB2KbqAWthnn31y/PjxRZchSZIkSYVYtmzZ45k5elvHNUVAHD9+PB0dHUWXIUmSJEmFiIhfbc9xDjGVJEmSJAEGREmSJElSmQFRkiRJkgQYECVJkiRJZQZESZIkSRJQZwExIq6JiN9ExE8H2B8RcWVErI6Izoh4Q61rlCRJkqRGVW/LXFwLXAVcP8D+twATy19HAp8qP0qSJElbtXh5NwuWrmLt+h7GjGpl9oxJzJw2tmGvK+2IugqImfmdiBi/lUNOB67PzAR+GBGjImK/zHysJgVKkiQ1iGYLLYuXd3PxLSvp2bgJgO71PVx8y0qAqr7uoq7be+1meo81NOpqiOl2GAusqdjuKrdJkiRpO/WGlu71PSQvhpbFy7trcu2jL72LCXO+xtGX3lWTawIsWLpqc0jr1bNxEwuWrmrI6xb5Hmt4q6sexO0Q/bRlvwdGnAecB7D//vtXsyZJkjTMNdvQw62FlkbtTVu7vmdQ7cP9ukW9xxr+hlsPYhcwrmK7DVjb34GZuTAz2zOzffTo0TUpTpIkDT9F9bQU2cNTj6Gl2saMah1U+3C/blHvcTMrqnd8qA23gLgEOKc8m+kbgQ3efyhJknZGsw09hOYMLbNnTKK1ZcQWba0tI5g9Y1JDXreo97hoRYW0RhrSW1cBMSJuAH4ATIqIroj484g4PyLOLx9yG/AwsBr4DPAXBZUqSZIaRLMNPYTmDC0zp43lkjOnMHZUKwGMHdXKJWdOqfpwy6KuW9R73KuIoFZkSCvyDz5Dra7uQczMs7exP4ELalSOJElqAmNGtdLdTyirxdDDIq4LL97vV+v7H2fPmLTFPYhQ29Ayc9rYQu6/K+K6Rb3HUNy9pkXed9lIQ3rrKiBKkiTVWlGhpRnDUpGhpRkVFYiLCmpFhrQi/+Az1AyIkiSpLhQ1o2dRoaVZw1JRoUW1U1RQKzKkFf0Hn6FkQJQkaSuKXGi6mZZeKHL5g95rNMvQQ6naigpqRYa0RvqDjwFRkqQBFBlairp2M947JGloFRXUig5pjfIHHwOiJEkDKDK0FHXtZrx3SNLQKjKoNUpIK5IBUZKkARQZWppt6YVGmuBBkkFtOKurdRAlSaonRa7ZVtS1i7pu0Wu2SZJKDIiSJA2gyNBS1LWLum5Ri4lLkrbkEFNJkgZQ9H00RVy76NdsIJSkYkVmFl1D1bW3t2dHR0fRZUiSJElSISJiWWa2b+s4h5hKkiRJkgCHmEqShokiF6yXJKlZGBAlSXWvyAXrJUlqJg4xlSTVva0t3i5JkoaOAVGSVPeKXLBekqRmYkCUJNW9IheslySpmRgQJUl1r8gF6yVJaiZOUiNJqntFLt4uSTutcxHcOR82dMHINpg+F6bOKrqqxuTPeqcZECVJw8LMaWMNhJKGn85FcOtFsLF8z/SGNaVtMLgMNX/WQ6LuhphGxMkRsSoiVkfEnH72HxARd0ZEZ0TcHRFtRdQpSUVavLyboy+9iwlzvsbRl97F4uXdRZckSerPnfNfDCy9NvaU2htZ5yK4YjLMG1V67FxU/WsW/bMu4jVXQV0FxIgYAVwNvAU4BDg7Ig7pc9jlwPWZORWYD1xS2yolqVi9awJ2r+8heXFNQEOiJNWhDV2Da28EvT15G9YA+WJPXrUDU5E/66JecxXUVUAEjgBWZ+bDmfkccCNwep9jDgHuLH//rX72S1JDc01ASRpGRg4w2G2g9qFUVI9WUT15Rf6si+69HEL1FhDHAmsqtrvKbZV+Ary9/P0ZwB4RsXcNapOkuuCagJI0jEyfCy19luRpaS21V1ORPVpF9eQV9bOGhuoprreAGP20ZZ/tvwOOi4jlwHFAN/D8S04UcV5EdEREx7p164a+UkkqSJFrAnrvoyQN0tRZcOqVMHIcEKXHU6+s/qQpRfZoFdWTV9TPGortvRxi9TaLaRcwrmK7DVhbeUBmrgXOBIiIVwJvz8wNfU+UmQuBhQDt7e19Q6YkDVuzZ0zi4ltWbjHMtBZrAvbe+9h73d57HwFnF5WkrZk6q/azaBbZozV97paziULtevKK+FlDsa95iNVbD+J9wMSImBARuwJnAUsqD4iIfSKit+6LgWtqXKMkFWrmtLFccuYUxo5qJYCxo1q55MwpVQ9p3vsoaUg0yEyPda/IHq0ie/KK0kCvua56EDPz+Yi4EFgKjACuycz7I2I+0JGZS4DjgUsiIoHvABcUVrCkprZ4eXdhC7cXsSag9z5K2mmuU1c7RfdoFdWTV6QGec11FRABMvM24LY+bXMrvr8ZuLnWdUlSpWYcbjlmVCvd/YTBWtz7KKkKOheV7kfb0FXqVZo+t9j74hrgH9Z1pffnWev3WMNe3QVESRoOtjbcslEDYlH3PkqqgqJ68oqe6bGIUFykBunRUm3V2z2IkjQsNONwy6LufZRUBc24Tl0DLWQuVZM9iJK0A5p1uGUR9z5KqoIi16kr6r44h7dK28UeREnaAbNnTKK1ZcQWbQ63lIaxZptZsxnXqSt6eKs0TNiDKEk7oLcXrahZTCUNoWacWbMZ16kb2VYeXtpPu6TNDIiStIMcbik1iGYcetiMM1wWveyDNEwYECVJUn0oaobJZh162GwzXDZjKJZ2gAFRkiQVr8hhnkUOPWy2ZReK1myhWNoBTlIjSZKKV9SyC1AKZS19ZiCuxdBDl12QVIcMiJIkqXhFDvMsambNIkOxJA3AIaaSJKl4Rc8wWcTQw2a991FSXbMHUdKwt3h5N0dfehcT5nyNoy+9i8XLu4suSdJgFTXMs0hFrUUoSVthQJQ0rC1e3s3Ft6yke30PCXSv7+HiW1YaEqXhpsgF1IvSjKFYUt1ziKmkYW3B0lX0bNy0RVvPxk0sWLrKNQqlHVXUzJrNNsOkyy5IqkMGREnD2tr1PYNql4aVIoJakctNNKNmC8WS6p5DTCUNa2NGtQ6qXRo2iloCwZk1JampGRAlDWuzZ0yitWXEFm2tLSOYPWNSQRVJQ6SooObMmpLU1BxiKmlY673PcMHSVaxd38OYUa3MnjHJ+w81/BUV1IpebkKSVCgDoqRhb+a0sQZCNZ6igtr0uVvegwjOrClJTaTuhphGxMkRsSoiVkfEnH727x8R34qI5RHRGRFvLaJOSZKqqqglEJpxuQlJ0mZ11YMYESOAq4ETgS7gvohYkpkPVBz2j8CizPxURBwC3AaMr3mxkiRVU5FLIDizpiQ1rboKiMARwOrMfBggIm4ETgcqA2ICryp/PxJYW9MKJUmqFYOaJKnG6i0gjgUqb7joAo7sc8w84OsR8ZfA7sCf1qY0SZIkSWps9XYPYvTTln22zwauzcw24K3A5yPiJa8jIs6LiI6I6Fi3bl0VSpUkSZKkxlJvAbELGFex3cZLh5D+ObAIIDN/AOwG7NP3RJm5MDPbM7N99OjRVSpXktTwOhfBFZNh3qjSY7UXqpckqUD1FhDvAyZGxISI2BU4C1jS55hHgekAEfE6SgHRLkJJ0tDrXFRa8mHDGiBLj7deZEiUJDWsugqImfk8cCGwFHiQ0myl90fE/Ig4rXzY3wIfioifADcA52Zm32GokiTtvDvnb7keIJS275xfTD2SJFVZvU1SQ2beRmnpisq2uRXfPwAcXeu6JElNaEPX4NolSRrm6qoHUZKkujKybXDtkiQNcwZESZIGMn0utLRu2dbSWmqXJKkBGRAlSRrI1Flw6pUwchwQpcdTr3TxeklSw6q7exAlSaorU2cZCCVJTcMeREnS8OB6hJIkVZ09iJKk+te7HmHvkhO96xGCvXuSJA0hexAlSfXP9QglSaoJA6Ikqf65HqEkSTVhQJQk1T/XI5QkqSYMiJKk+ud6hJIk1YQBUZJU/1yPUJKkmnAWU0nS8OB6hJIkVZ09iJIkSZIkwIAoSZIkSSpziKmkIbF4eTcLlq5i7foexoxqZfaMScycNrbosjTUOheV1h7c0FWaQXT6XId9SpLUQAyIknba4uXdXHzLSno2bgKge30PF9+yEsCQ2Eg6F8GtF724YP2GNaVtMCRKktQgHGIqaactWLpqczjs1bNxEwuWriqooibQuQiumAzzRpUeOxdV/5p3zn8xHPba2FNqlyRJDcEeREk7be36nkG1aycV1ZO3oWtw7ZIkadixB1FqMIuXd3P0pXcxYc7XOPrSu1i8vLvq1xwzqnVQ7dpJRfXkjWwbXLskSRp2DIhSA+m9F7B7fQ/Ji/cCVjskzp4xidaWEVu0tbaMYPaMSVW9btMqqidv+lxo6RP6W1pL7ZIkqSHUXUCMiJMjYlVErI6IOf3svyIiVpS/fh4R64uoU6pHRd0LOHPaWC45cwpjR7USwNhRrVxy5hQnqKmWonryps6CU6+EkeOAKD2eeqUT1EiS1EDq6h7EiBgBXA2cCHQB90XEksx8oPeYzPzriuP/EphW80KlOlXkvYAzp401ENbK9Llb3oMItevJmzrLQChJUgOrtx7EI4DVmflwZj4H3AicvpXjzwZuqEll0jDgvYBNwp48SZJUJXXVgwiMBdZUbHcBR/Z3YEQcAEwA7qpBXdKwMHvGpC3WIwTvBWxY9uRJkqQqqLeAGP205QDHngXcnJmb+tsZEecB5wHsv//+Q1OdVOd6h3guWLqKtet7GDOqldkzJjn0U5IkSdul3gJiFzCuYrsNWDvAsWcBFwx0osxcCCwEaG9vHyhkSg3HewFrqHNRaWmJDV2lCWKmz7VXT5IkDWv1FhDvAyZGxASgm1IIfHffgyJiErAn8IPalidJZUUtVi9JklRFVZmkJiIujIg9B/u8zHweuBBYCjwILMrM+yNifkScVnHo2cCNmWnPoKRiFLVYvSRJUhVVqwfxNZSWqPgxcA2wdHvDXGbeBtzWp21un+15Q1SnJO2YoharlyRJqqKq9CBm5j8CE4HPAucCD0XEv0TEa6txPUmquaIWq5ckSaqiqq2DWO4x/HX563lK9wzeHBGXVeuaklQz0+eWFqevVKvF6iVJkqqkKkNMI+Ii4P3A48B/AbMzc2NEvAx4CPj7alxXkmqmdyIaZzGVJEkNpFr3IO4DnJmZv6pszMwXIuKUKl1TkmrLxeolSVKDqdYQ09uAJ3s3ImKPiDgSIDMfrNI1JUmSJEk7oVoB8VPA7yq2f19ukyRJkiTVqWoFxKhc1iIzX6B6w1klSZIkSUOgWgHx4Yi4KCJayl9/BTxcpWtJkiRJkoZAtQLi+cCbgG6gCzgSOK9K15IkSZIkDYGqDPvMzN8AZ1Xj3JIkSZKk6qjWOoi7AX8OvB7Yrbc9M/+sGteTJEmSJO28ag0x/TzwGmAG8G2gDXi6SteSJEmSJA2BagXEgzLzn4DfZ+Z1wNuAKVW6lqR60LkIrpgM80aVHjsXFV2RJEmSBqlaS09sLD+uj4jJwK+B8VW6lqSidS6CWy+CjT2l7Q1rStsAU2cVV5ckSZIGpVo9iAsjYk/gH4ElwAPAv1bpWpKKduf8F8Nhr409pfZasPdSkiRpSAx5D2JEvAz4bWY+BXwHOHCoryGpzmzoGlz7ULL3UpIkacgMeQ9iZr4AXDjU55VUx0a2Da59KBXdeylJktRAqjXE9BsR8XcRMS4i9ur9qtK1JBVt+lxoad2yraW11F5tRfZeSpIkNZhqTVLTu97hBRVticNNpcbUO5TzzvmlYDayrRQOazHEc2RbaVhpf+2SJEkalKoExMycsKPPjYiTgf8fGAH8V2Ze2s8xs4B5lELnTzLz3Tt6PUlDZOqsYu75mz53y3sQoXa9l5IkSQ2mKgExIs7prz0zr9/G80YAVwMnAl3AfRGxJDMfqDhmInAxcHRmPhURrx66yqWhs3h5NwuWrmLt+h7GjGpl9oxJzJw2tuiyGk+RvZeSJEkNplpDTA+v+H43YDrwY2CrARE4AlidmQ8DRMSNwOmUlsno9SHg6vIsqWTmb4aqaGmoLF7ezcW3rKRn4yYAutf3cPEtKwEMidVQVO+lJElSg6nWENO/rNyOiJHA57fjqWOBypuJuoAj+xzzR+Vzfo/SMNR5mXnHjlcrDb0FS1dtDoe9ejZuYsHSVQZESZIk1a1q9SD29Qdg4nYcF/20ZZ/tXcrnOh5oA74bEZMzc/0WJ4o4DzgPYP/99x9svdJOWbu+Z1DtkiRJUj2o1j2It/JisHsZcAiwaDue2gWMq9huA9b2c8wPM3Mj8MuIWEUpMN5XeVBmLgQWArS3t/cNmVJVjRnVSnc/YXDMqNZ+jpYkSZLqQ7V6EC+v+P554FeZuT2Lkt0HTIyICUA3cBbQd4bSxcDZwLURsQ+lIacP73zJ0tCZPWPSFvcgArS2jGD2jEkFViVJkiRtXbUC4qPAY5n5DEBEtEbE+Mx8ZGtPysznI+JCYCml+wuvycz7I2I+0JGZS8r7ToqIB4BNwOzMfKJKr0PaIb33GTqLqSRJkoaTyBz60ZcR0QG8KTOfK2/vCnwvMw/f+jOro729PTs6Ooq4tCRJkiQVLiKWZWb7to57WZWuv0tvOAQof79rla4lSZIkSRoC1QqI6yLitN6NiDgdeLxK15IkSZIkDYFq3YN4PvDFiLiqvN0FnFOla0mSJEmShkBVAmJm/gJ4Y0S8ktJ9jk9X4zqS+tG5CO6cDxu6YGQbTJ8LU2cVXZUkSZKGgaoMMY2If4mIUZn5u8x8OiL2jIh/rsa1JFXoXAS3XgQb1gBZerz1olK7JEmStA3VugfxLZm5vncjM58C3lqla0nqded82NizZdvGnlK7JEmStA3VCogjIuLlvRsR0Qq8fCvHSxoKG7oG1y5JkiRVqNYkNV8A7oyIz5W3PwBcV6VrSeo1sq08vLSfdkmSJGkbqtKDmJmXAf8MvA44BLgDOKAa15JUYfpcaGndsq2ltdQuSZIkbUO1hpgC/Bp4AXg7MB14sIrXkgSl2UpPvRJGjgOi9Hjqlc5iKkmSpO0ypENMI+KPgLOAs4EngJsoLXPx5qG8jqStmDrLQChJkqQdMtT3IP4M+C5wamauBoiIvx7ia0jbbfHybhYsXcXa9T2MGdXK7BmTmDltbNFlSZIkSXVpqIeYvp3S0NJvRcRnImI6EEN8DWm7LF7ezcW3rKR7fQ8JdK/v4eJbVrJ4eXfRpUmSJEl1aUgDYmZ+OTPfBRwM3A38NbBvRHwqIk4aymtJ27Jg6Sp6Nm7aoq1n4yYWLF1VmwI6F8EVk2HeqNKji9VLkiSpzlVrFtPfZ+YXM/MUoA1YAcypxrWkgaxd3zOo9iHVuQhuvai85ESWHm+9yJAoSZKkulbNWUwByMwnM/M/M/OEal9LqjRmVOug2ofUnfNhY58gurGn1C5JkiTVqaoHRKkos2dMorVlxBZtrS0jmD1jUvUvvqFrcO2SJElSHRjqWUylutE7W2khs5iObCsPL+2nXZIkSapTBkQ1tJnTxhazrMX0uaV7DiuHmba0ltolSZKkOlV3Q0wj4uSIWBURqyPiJRPbRMS5EbEuIlaUvz5YRJ3SVk2dBadeCSPHAVF6PPVKF7CXJElSXaurHsSIGAFcDZwIdAH3RcSSzHygz6E3ZeaFNS9Qw0/notLEMBu6SsM7p8+tXUibOstAKEmSpGGl3noQjwBWZ+bDmfkccCNwesE1abhyqQlJkiRpUOotII4FKmf26Cq39fX2iOiMiJsjYlxtStOw41ITkiRJ0qDUW0CMftqyz/atwPjMnAp8E7iu3xNFnBcRHRHRsW7duiEuU8OCS01IkiRJg1JvAbELqOwRbAPWVh6QmU9k5rPlzc8Af9zfiTJzYWa2Z2b76NGjq1Ks6txAS0q41IQkSZLUr3oLiPcBEyNiQkTsCpwFLKk8ICL2q9g8DXiwhvVpOJk+t7S0RCWXmpAkSZIGVFezmGbm8xFxIbAUGAFck5n3R8R8oCMzlwAXRcRpwPPAk8C5hRWs7bZ4eXftF6zvnUG0qFlMJUmSpGEmMvve4td42tvbs6Ojo+gymtbi5d1cfMtKejZu2tzW2jKCS86cUswi9pIkSVKTiYhlmdm+rePqbYipGtCCpau2CIcAPRs3sWDpqoIqkiRJktQfA6Kqbu36nkG1S5IkSSqGAVFVN2ZU66DaJUmSJBXDgKiqmz1jEq0tI7Zoa20ZwewZkwqqSJIkSVJ/6moWUzWm3oloaj6LqSRJkqRBMSCqJmZOG2sglCRJkuqcQ0wlSZIkSYABUZIkSZJUZkCUJEmSJAEGREmSJElSmQFRkiRJkgQYECVJkiRJZQZESZIkSRJgQJQkSZIklRkQJUmSJEmAAVGSJEmSVLZL0QWodhYv72bB0lWsXd/DmFGtzJ4xiZnTxhZdliRJkqQ6YUBsEouXd3PxLSvp2bgJgO71PVx8y0oAQ6IkSZIkwCGmTWPB0lWbw2Gvno2bWLB0VUEVSZIkSao3BsQmsXZ9z6DaJUmSJDWfuguIEXFyRKyKiNURMWcrx70jIjIi2mtZ33A1ZlTroNqHXOciuGIyzBtVeuxcVJvrSpIkSdpudRUQI2IEcDXwFuAQ4OyIOKSf4/YALgLurW2Fw9fsGZNobRmxRVtrywhmz5hU/Yt3LoJbL4INa4AsPd56kSFRkiRJqjN1FRCBI4DVmflwZj4H3Aic3s9x/we4DHimlsUNZzOnjeWSM6cwdlQrAYwd1colZ06pzQQ1d86HjX2Gsm7sKbVLkiRJqhv1NovpWGBNxXYXcGTlARExDRiXmV+NiL+rZXHD3cxpY4uZsXRD1+DaJUmSJBWi3noQo5+23Lwz4mXAFcDfbvNEEedFREdEdKxbt24IS9SgjWwbXLskSZKkQtRbQOwCxlVstwFrK7b3ACYDd0fEI8AbgSX9TVSTmQszsz0z20ePHl3FkrVN0+dCS5/JcFpaS+2SJEmS6ka9BcT7gIkRMSEidgXOApb07szMDZm5T2aOz8zxwA+B0zKzo5hytV2mzoJTr4SR44AoPZ56ZaldkiRJUt2oq3sQM/P5iLgQWAqMAK7JzPsjYj7QkZlLtn4GbVXnotLEMBu6SsM7p8+tXUibOstAKEmSJNW5ugqIAJl5G3Bbn7Z+xyJm5vG1qKkh9C410TubaO9SE2BwkyRJkgTU3xBTVYtLTUiSJEnaBgNis3CpCUmSJEnbYEBsFi41IUmSJGkbDIjNwqUmJEmSJG2DAbFZuNSEJEmSpG2ou1lMVUUuNSFJkiRpK+xBlCRJkiQBBkRJkiRJUpkBUZIkSZIEGBAlSZIkSWUGREmSJEkSYECUJEmSJJUZECVJkiRJgAFRkiRJklRmQJQkSZIkAQZESZIkSVKZAVGSJEmSBBgQJUmSJEllBkRJkiRJElCHATEiTo6IVRGxOiLm9LP//IhYGRErIuKeiDikiDqsSxgzAAAgAElEQVQlSZIkqdHUVUCMiBHA1cBbgEOAs/sJgF/KzCmZeRhwGfCJGpcpSZIkSQ2prgIicASwOjMfzszngBuB0ysPyMzfVmzuDmQN65MkSZKkhrVL0QX0MRZYU7HdBRzZ96CIuAD4G2BX4ITalCZJkiRJja3eehCjn7aX9BBm5tWZ+Vrgo8A/9nuiiPMioiMiOtatWzfEZUqSJElS46m3gNgFjKvYbgPWbuX4G4GZ/e3IzIWZ2Z6Z7aNHjx7CEiVJkiSpMdVbQLwPmBgREyJiV+AsYEnlARExsWLzbcBDNaxPkiRJkhpWXd2DmJnPR8SFwFJgBHBNZt4fEfOBjsxcAlwYEX8KbASeAt5fXMWSJEmS1DjqKiACZOZtwG192uZWfP9XNS9KkiRJkppAvQ0xlSRJkiQVxIAoSZIkSQIMiJIkSZKksrq7B1GSJEnNZePGjXR1dfHMM88UXYo07O222260tbXR0tKyQ883IBZg8fJuFixdxdr1PYwZ1crsGZOYOW1s0WVJkiQVoquriz322IPx48cTEUWXIw1bmckTTzxBV1cXEyZM2KFzOMS0xhYv7+biW1bSvb6HBLrX93DxLStZvLy76NIkSZIK8cwzz7D33nsbDqWdFBHsvffeO9Ubb0CssQVLV9GzcdMWbT0bN7Fg6aqCKpIkSSqe4VAaGjv735IBscbWru8ZVLskSZKq78/+7M949atfzeTJkwc8JjO56KKLOOigg5g6dSo//vGPN++77rrrmDhxIhMnTuS6667b5vXuvvtuTjnllCGpfUdce+21rF27trDrD7U77riDSZMmcdBBB3HppZf2e8wnPvEJDjnkEKZOncr06dP51a9+tXnfiBEjOOywwzjssMM47bTTtnm9Rx55ZKuflWpbvHgxDzzwQFXObUCssTGjWgfVLkmSpOo799xzueOOO7Z6zO23385DDz3EQw89xMKFC/nIRz4CwJNPPsnHPvYx7r33Xn70ox/xsY99jKeeemqH6nj++ed36HmD1UgBcdOmTVxwwQXcfvvtPPDAA9xwww39hqdp06bR0dFBZ2cn73jHO/j7v//7zftaW1tZsWIFK1asYMmSJTtcS63ePwNiA5k9YxKtLSO2aGttGcHsGZMKqkiSJGl4Wby8m6MvvYsJc77G0ZfeNSRzORx77LHstddeWz3mK1/5Cueccw4RwRvf+EbWr1/PY489xtKlSznxxBPZa6+92HPPPTnxxBP7DZt33HEHBx98MMcccwy33HLL5vZ58+Zx3nnncdJJJ3HOOefwzDPP8IEPfIApU6Ywbdo0vvWtbwGlUHf66adz8sknM2nSJD72sY9tPscnPvEJJk+ezOTJk/nkJz8JvLSX6/LLL2fevHncfPPNdHR08J73vIfDDjuMnp4aj2TrXARXTIZ5o0qPnYt26nQ/+tGPOOiggzjwwAPZddddOeuss/jKV77ykuPe/OY384pXvAKAN77xjXR1dQ3qOsuWLePQQw/lqKOO4uqrr97cfu211/LOd76TU089lZNOOonMZPbs2UyePJkpU6Zw0003AaVe42OPPZYzzjiDQw45hPPPP58XXngBgBtuuIEpU6YwefJkPvrRj24+9ytf+crN3998882ce+65fP/732fJkiXMnj2bww47jF/84heDeh3b4iymNdY7W6mzmEqSJA1e74R/vXM69E74B1T931Pd3d2MGzdu83ZbWxvd3d0Dtld65pln+NCHPsRdd93FQQcdxLve9a4t9i9btox77rmH1tZW/u3f/g2AlStX8rOf/YyTTjqJn//850ApDP30pz/lFa94BYcffjhve9vbiAg+97nPce+995KZHHnkkRx33HHsueee/b6Od7zjHVx11VVcfvnltLe3D8nPZrt1LoJbL4KN5VC6YU1pG2DqrB06ZX8//3vvvXerz/nsZz/LW97yls3bzzzzDO3t7eyyyy7MmTOHmTNnvuQ5H/jAB/j3f/93jjvuOGbPnr3Fvh/84Ad0dnay11578T//8z+sWLGCn/zkJzz++OMcfvjhHHvssUDp/XvggQc44IADOPnkk7nlllt405vexEc/+lGWLVvGnnvuyUknncTixYv7rQHgTW96E6eddhqnnHIK73jHO7b757S9DIgFmDltrIFQkiRpB2xtwr9q//sqM1/SFhEDtlf62c9+xoQJE5g4cSIA733ve1m4cOHm/aeddhqtraVbju655x7+8i//EoCDDz6YAw44YHNAPPHEE9l7770BOPPMM7nnnnuICM444wx23333ze3f/e53t+teupq7c/6L4bDXxp5S+w4GxO35+Vf6whe+QEdHB9/+9rc3tz366KOMGTOGhx9+mBNOOIEpU6bw2te+dvP+DRs2sH79eo477jgA3ve+93H77bdv3t/bgwyl9+/ss89mxIgR7Lvvvhx33HHcd999vOpVr+KII47gwAMPBODss8/mnnvuoaWlheOPP57Ro0cD8J73vIfvfOc7AwbEanOIaRGGuFtdkiSpWRQ54V9bWxtr1qzZvN3V1cWYMWMGbO9ra6GlN9xB/4FnoHMMFFABdtlll81DGIGdWvpgyGwYYFjnQO3bYXt//gDf/OY3+fjHP86SJUt4+ctfvrm99/gDDzyQ448/nuXLl2/xvMys+fvX9/havX8GxFrr7VbfsAbIF7vVDYmSJEnbVOSEf6eddhrXX389mckPf/hDRo4cyX777ceMGTP4+te/zlNPPcVTTz3F17/+dWbMmLHFcw8++GB++ctfbr5f7IYbbhjwOsceeyxf/OIXAfj5z3/Oo48+yqRJpfkqvvGNb/Dkk0/S09PD4sWLOfroozn22GNZvHgxf/jDH/j973/Pl7/8Zf7kT/6Efffdl9/85jc88cQTPPvss3z1q1/dfI099tiDp59+eqh/RNs2sm1w7dvh8MMP56GHHuKXv/wlzz33HDfeeGO/vafLly/nwx/+MEuWLOHVr3715vannnqKZ599FoDHH3+c733vexxyyCFbPHfUqFGMHDmSe+65B2Dz+9OfY489lptuuolNmzaxbt06vvOd73DEEUcApSGmv/zlL3nhhRe46aabOOaYYzjyyCP59re/zeOPP86mTZu44YYbNvdU7rvvvjz44IO88MILfPnLX958jWq+fwbEWttat7okSZK2qloT/p199tkcddRRrFq1ira2Nj772c8C8OlPf5pPf/rTALz1rW/lwAMP5KCDDuJDH/oQ//Ef/wHAXnvtxT/90z9x+OGHc/jhhzN37tyXTHiz2267sXDhQt72trdxzDHHcMABBwxYy1/8xV+wadMmpkyZwrve9S6uvfbazb1dxxxzDO973/s47LDDePvb3057eztveMMbOPfcczniiCM48sgj+eAHP8i0adNoaWlh7ty5HHnkkZxyyikcfPDBm69x7rnncv7559d+kprpc6GlT5hvaS2176BddtmFq666ihkzZvC6172OWbNm8frXvx6AuXPnbp6VdPbs2fzud7/jne985xbLWTz44IO0t7dz6KGH8uY3v5k5c+a8JCACfO5zn+OCCy7gqKOO2jwcuD9nnHEGU6dO5dBDD+WEE07gsssu4zWveQ0ARx11FHPmzGHy5MlMmDCBM844g/32249LLrmEN7/5zRx66KG84Q1v4PTTTwfg0ksv5ZRTTuGEE05gv/3223yNs846iwULFjBt2rQhn6Qmttal2Sja29uzo6Oj6DJK5o0C+vuZB8xbX+tqJEmSCvfggw/yute9bruPX7y8uykn/Lv22mvp6OjgqquuKrqUndO5qNQ5sqGr1HM4fe4O3384nNx9991cfvnlW/TkVkt//01FxLLM3OasRE5SU2sj28rDS/tplyRJ0jY54d8wN3VWUwTC4cohprVWhW51SZIkNb5zzz13+PceNrHjjz++Jr2HO8uAWGtTZ8GpV8LIcUCUHk+90r+iSJIkSSqcQ0yLYLe6JEnSFra1jICk7bOzc8zUXQ9iRJwcEasiYnVEzOln/99ExAMR0RkRd0bEwFNASZIkqe7ttttuPPHEEzv9D1up2WUmTzzxBLvtttsOn6OuehAjYgRwNXAi0AXcFxFLMvOBisOWA+2Z+YeI+AhwGfCu2lcrSZKkodDW1kZXVxfr1q0ruhRp2Nttt91oa9vxCTDrKiACRwCrM/NhgIi4ETgd2BwQM/NbFcf/EHhvTSuUJEnSkGppaWHChAlFlyGJ+htiOhaoXAOiq9w2kD8Hbq9qRZIkSZLUJOqtB7G/O5P7HYweEe8F2oHjBth/HnAewP777z9U9UmSJElSw6q3HsQuYFzFdhuwtu9BEfGnwP8LnJaZz/Z3osxcmJntmdk+evToqhQrSZIkSY0k6mm2qIjYBfg5MB3oBu4D3p2Z91ccMw24GTg5Mx/azvOuA3419BXvtH2Ax4suQg3Lz5eqyc+XqsnPl6rNz5iqqV4/Xwdk5jZ7zuoqIAJExFuBTwIjgGsy8+MRMR/oyMwlEfFNYArwWPkpj2bmaQWVu1MioiMz24uuQ43Jz5eqyc+XqsnPl6rNz5iqabh/vurtHkQy8zbgtj5tcyu+/9OaFyVJkiRJTaDe7kGUJEmSJBXEgFishUUXoIbm50vV5OdL1eTnS9XmZ0zVNKw/X3V3D6IkSZIkqRj2IEqSJEmSAANiISLi5IhYFRGrI2JO0fWo8UTEIxGxMiJWRERH0fVoeIuIayLiNxHx04q2vSLiGxHxUPlxzyJr1PA1wOdrXkR0l3+HrSjPcC4NWkSMi4hvRcSDEXF/RPxVud3fYdppW/l8DevfYQ4xrbGIGEFprccTgS5Kaz2enZkPFFqYGkpEPAK0Z2Y9rsGjYSYijgV+B1yfmZPLbZcBT2bmpeU/dO2ZmR8tsk4NTwN8vuYBv8vMy4usTcNfROwH7JeZP46IPYBlwEzgXPwdpp20lc/XLIbx7zB7EGvvCGB1Zj6cmc8BNwKnF1yTJA0oM78DPNmn+XTguvL311H6H6I0aAN8vqQhkZmPZeaPy98/DTwIjMXfYRoCW/l8DWsGxNobC6yp2O6iAT5IqjsJfD0ilkXEeUUXo4a0b2Y+BqX/QQKvLrgeNZ4LI6KzPATV4X/aaRExHpgG3Iu/wzTE+ny+YBj/DjMg1l700+Y4Xw21ozPzDcBbgAvKQ7gkabj4FPBa4DDgMeDfii1Hw11EvBL4H+D/yczfFl2PGks/n69h/TvMgFh7XcC4iu02YG1BtahBZeba8uNvgC9TGtosDaX/Ld970XsPxm8KrkcNJDP/NzM3ZeYLwGfwd5h2QkS0UPrH+xcz85Zys7/DNCT6+3wN999hBsTauw+YGBETImJX4CxgScE1qYFExO7lG6WJiN2Bk4Cfbv1Z0qAtAd5f/v79wFcKrEUNpvcf7mVn4O8w7aCICOCzwIOZ+YmKXf4O004b6PM13H+HOYtpAcpT3X4SGAFck5kfL7gkNZCIOJBSryHALsCX/IxpZ0TEDcDxwD7A/wL/H7AYWATsDzwKvDMznWhEgzbA5+t4SkOzEngE+HDv/WLSYETEMcB3gZXAC+Xmf6B0n5i/w7RTtvL5Opth/DvMgChJkiRJAhxiKkmSJEkqMyBKkiRJkgADoiRJkiSpzIAoSZIkSQIMiJIkSZKkMgOiJEnbKSI2RcSKiq85Q3ju8RExrNbKkiQ1nl2KLkCSpGGkJzMPK7oISZKqxR5ESZJ2UkQ8EhH/GhE/Kn8dVG4/ICLujIjO8uP+5fZ9I+LLEfGT8tebyqcaERGfiYj7I+LrEdFa2IuSJDUlA6IkSduvtc8Q03dV7PttZh4BXAV8stx2FXB9Zk4FvghcWW6/Evh2Zh4KvAG4v9w+Ebg6M18PrAfeXuXXI0nSFiIzi65BkqRhISJ+l5mv7Kf9EeCEzHw4IlqAX2fm3hHxOLBfZm4stz+WmftExDqgLTOfrTjHeOAbmTmxvP1RoCUz/7n6r0ySpBJ7ECVJGho5wPcDHdOfZyu+34RzBUiSasyAKEnS0HhXxeMPyt9/Hzir/P17gHvK398JfAQgIkZExKtqVaQkSVvjXyYlSdp+rRGxomL7jszsXeri5RFxL6U/vp5dbrsIuCYiZgPrgA+U2/8KWBgRf06pp/AjwGNVr16SpG3wHkRJknZS+R7E9sx8vOhaJEnaGQ4xlSRJkiQB9iBKkiRJksrsQZQkqSwixkdERoT36EuSmpIBUZLUMCJiaUTM76f99Ij4tcFPkqStMyBKkhrJtcD7IiL6tL8P+GJmPl/7koZGlPj/bUlSVfk/GklSI1kM7AX8SW9DROwJnAJcX95+W0Qsj4jfRsSaiJi3vSePiDkR8YuIeDoiHoiIM/rs/1BEPFix/w3l9nERcUtErIuIJyLiqnL7vIj4QsXztxjiGhF3R8THI+J7wB+AAyPiAxXXeDgiPtynhtMjYkX59f0iIk6OiHdGxLI+x/1tRCze3tcuSWoOBkRJUsPIzB5gEXBORfMs4GeZ+ZPy9u/L+0cBbwM+EhEzt/MSv6AUPkcCHwO+EBH7AUTEO4F55XO/CjgNeCIiRgBfBX4FjAfGAjcO4mW9DzgP2KN8jt9QCryvorSu4hUVQfQISkF4dvn1HQs8AiwBJkTE6yrO+17g84OoQ5LUBAyIkqRGcx3wzohoLW+fU24DIDPvzsyVmflCZnYCNwDHbc+JM/O/M3Nt+bk3AQ8BR5R3fxC4LDPvy5LVmfmr8v4xwOzM/H1mPpOZ9wzi9Vybmfdn5vOZuTEzv5aZvyhf49vA13mxx/TPgWsy8xvlGrsz82eZ+SxwE6VQSES8nlJY/eog6pAkNQEDoiSpoZTD1zrg9Ig4EDgc+FLv/og4MiK+VR7uuQE4H9hne84dEeeUh2+uj4j1wOSK546j1MPY1zjgVztx/+OaPjW8JSJ+GBFPlmt463bUAKWQ/O7y/ZnvAxaVg6MkSZsZECVJjeh6Sj2H7wO+npn/W7HvS5SGXI7LzJHAp4G+k9q8REQcAHwGuBDYOzNHAT+teO4a4LX9PHUNsP8AM6j+HnhFxfZr+jlm84LFEfFy4H+Ay4F9yzXcth01kJk/BJ6j1Nv4bhxeKknqhwFRktSIrgf+FPgQFcNLy/YAnszMZ8r37L17O8+5O6Wwtg4gIj5AqQex138BfxcRf1yecfSgcqj8EfAYcGlE7B4Ru0XE0eXnrACOjYj9I2IkcPE2atgVeHm5hucj4i3ASRX7Pwt8ICKmR8TLImJsRBxcsf964Crg+UEOc5UkNQkDoiSp4WTmI8D3KYW6JX12/wUwPyKeBuZSmtRme875APBvwA+A/wWmAN+r2P/fwMcp9VA+TXlG1czcBJwKHAQ8CnQB7yo/5xuU7g3sBJaxjXsCM/Np4KJyzU9RCrdLKvb/iPLENcAG4NvAARWn+DylUGvvoSSpX5GZ2z5KkiQNe+WJe34DvCEzHyq6HklS/bEHUZKk5vER4D7DoSRpIP3dMC9JkhpMRDxCaTKb7V3zUZLUhBxiKkmSJEkCHGIqSZIkSSozIEqSJEmSgCa5B3GfffbJ8ePHF12GJEmSJBVi2bJlj2fm6G0d1xQBcfz48XR0dBRdhiRJkiQVIiJ+tT3HOcRUkiRJkgQYECVJkiRJZQZESZIkSRJgQJQkSZIklVU1IEbEyRGxKiJWR8ScfvafHxErI2JFRNwTEYeU28dHRE+5fUVEfLriOX9cfs7qiLgyIqKar0GSJEmSmkXVAmJEjACuBt4CHAKc3RsAK3wpM6dk5mHAZcAnKvb9IjMPK3+dX9H+KeA8YGL56+RqvQZJkiRJaibV7EE8AlidmQ9n5nPAjcDplQdk5m8rNncHcmsnjIj9gFdl5g8yM4HrgZlDW7YkSU2ucxFcMRnmjSo9di4quiJJUo1Ucx3EscCaiu0u4Mi+B0XEBcDfALsCJ1TsmhARy4HfAv+Ymd8tn7OrzznHDnHdkiQ1r85FcOtFsLGntL1hTWkbYOqs4uqSJNVENXsQ+7s38CU9hJl5dWa+Fvgo8I/l5seA/TNzGqXw+KWIeNX2nhMgIs6LiI6I6Fi3bt0OvQBJkprOnfNfDIe9NvaU2iVJDa+aAbELGFex3Qas3crxN1IeLpqZz2bmE+XvlwG/AP6ofM627TlnZi7MzPbMbB89evQOvwhJkprKhq7BtUuSGko1A+J9wMSImBARuwJnAUsqD4iIiRWbbwMeKrePLk9yQ0QcSGkymocz8zHg6Yh4Y3n20nOAr1TxNUiS1FxGtg2uXZLUUKoWEDPzeeBCYCnwILAoM++PiPkRcVr5sAsj4v6IWEFpKOn7y+3HAp0R8RPgZuD8zHyyvO8jwH8Bqyn1LN5erdcgSVLTmT4XWlq3bGtpLbVLkhpelCYDbWzt7e3Z0dFRdBmSJA0PnYtK9xxu6Cr1HE6f6wQ1kjTMRcSyzGzf1nHVnMVUkiQNR1NnGQglqUlV8x5ESZIkSdIwYkCUJEmSJAEGREmSJElSmQFRkiTVh85FcMVkmDeq9Ni5qOiKVA2+z1Jdc5IaSZJUvM5FcOtFsLGntL1hTWkbnDCnkfg+S3XPHkRJklS8O+e/GBp6bewptatx+D5Ldc+AKEmSireha3DtGp58n6W6Z0CUJEnFG9k2uHYNT77PUt0zIEqSpOJNnwstrVu2tbSW2tU4fJ+lumdAlCRJxZs6C069EkaOA6L0eOqVTlzSaHyfpboXmVl0DVXX3t6eHR0dRZchSZIkSYWIiGWZ2b6t4+xBlCRJkiQBBkRJkiRJUpkBUZIkSZIEGBAlSZIkSWUGREmSJEkSYECUJEmSJJUZECXp/7Z372F21eXB9793TmQiNsMhcsgEDJI3NCYxaQdQoYAiBBRCpBiJB0CtgJWX9vF6IuGt0sCrDymh6kPhaaVV0IpApBBjFaLloE2LmAlJEw5GEBRmgiUBgqITCMn9/LH3xJ0wk8xkZs3KzP5+rmtfe697HfZv7ays2ff+nSRJ0uCwehF8cTLMb6w8r15UdokGnGFlF0CSJEmSem31IvjOxbC5vbL84tOVZYCps8sr1wBjDaIkSZKkge/uK36fHHbY3F6Jq9usQZQkSapDi1e2sXDpWtZtbOfgxgbmzpjIrOljyy6WtPtebO1ZXJ0qtAYxIk6JiLUR8XhEzOtk/YURsSYiVkXEsoiYVI2fFBErqutWRMQ7a/a5r3rMVdXHG4o8B0mSpMFm8co2Lr19DW0b20mgbWM7l96+hsUr28oumrT7Rjf1LK5OFZYgRsRQ4DrgVGASMKcjAazxzcyckpnTgKuAL1TjG4DTM3MKcC7wzzvs98HMnFZ9PFvUOUiSJA1GC5eu5aQtP2TZiIt5Yq8PsGzExZy05YcsXLq27KJJu+/Ey2B4w/ax4Q2VuLqtyBrEo4DHM/OJzHwFuAU4o3aDzPx1zeLrgKzGV2bmumr8YWBkROxVYFklSZLqRvOvf8CC4f9E05ANDAloGrKBBcP/ieZf/6Dsokm7b+psOP0aGD0OiMrz6dc4QE0PFdkHcSzwdM1yK3D0jhtFxCeBTwEjgHfuuB74U2BlZr5cE7shIrYA/wJ8LjOzz0otSZI0yF064luM4pXtYqPiFS4d8S3gynIKJfWFqbNNCHupyBrE6CT2mkQuM6/LzDcBlwCf2e4AEW8G/ga4oCb8wWrT0z+pPj7c6ZtHnB8RLRHRsn79+t08BUmSpMHnADb0KN6XFq9s45gF9zB+3nc5ZsE99nuU9jBFJoitwLia5SZgXRfbQqUJ6qyOhYhoAu4AzsnMn3fEM7Ot+vwb4JtUmrK+RmZen5nNmdk8ZsyY3T4JSZKkwSa6GLSjq3hfqdvBcZy8XQNIkQnicmBCRIyPiBHA2cCS2g0iYkLN4nuAx6rxRuC7wKWZ+R812w+LiP2rr4cDpwEPFXgOkiRJg09Jg3ksXLqW9s1btou1b94yuAfH6Zi8/cWngfz95O0midpDFZYgZuarwEXAUuBRYFFmPhwRV0TEzOpmF0XEwxGxiko/xHM74sDhwGd3mM5iL2BpRKwGVgFtwD8WdQ6SJEmDUkmDeazb2N6j+KDg5O0aYIocpIbM/B7wvR1il9W8/osu9vsc8LkuDvvHfVZASZKkelXCYB4HNzbQ1kkyeHBjQydbDxJO3q4BpsgmppIkSdI2c2dMpGH40O1iDcOHMnfGxJJK1A+cvF0DjAmiJEmS+sWs6WO58swpjG1sIICxjQ1ceeYUZk0fW3bRiuPk7RpgCm1iKkmSJNWaNX3s4E4Id9TRjPfuKyrNSkc3VZJD5+rTHsoEUZIkSSqSk7drALGJqSRJkiQJMEGUJGnP5eTakqR+ZhNTSZL2RB2Ta3fMn9YxuTbYVE3aTYtXtrFw6VrWbWzn4MYG5s6YWF/9IaVusAZRkqQ9kZNrS31q8co2Lr19DW0b20mgbWM7l96+hsUr28oumrRHMUGUJGlP5OTaUp9auHQt7Zu3bBdr37yFhUvXllQiac9kE1NJ2l2rFzlsuYozuqnSrLSzeMFshtd//Kz7z7qN7T2KS/XKGkRJ2h0d/cNefBrI3/cPcxAR9ZWSJte2GV7/8bPuXwc3NvQoLtUrE0RJ2h32D1PRps6G06+B0eOAqDyffk3htdQ2w+s/ftb9a+6MiTQMH7pdrGH4UObOmFhSiaQ9k01MJWl32D9M/aGEybVthtd//Kz7V0fTXZv0SjtngihJu6PE/mF1yz6f/eLgxgbaOklQbIbX9/ys+9+s6WPrKyH0vtm/BsnnbRNTSdodJfUPq1v2+ew3NsPrP37WKlTZ983Vi+CLk2F+Y+V5sN+vy/68+1BkZtllKFxzc3O2tLSUXQxJg80g+aVwQPji5C5qbMfB/3io/8szyJU1smY9juhZj+dcj0r5dy7zvtmRLNX21R/e0C/9qEszAP5ORcSKzGze5XYmiJKkPd78RqCzv1cB8zf2d2lUgI4RPWsHbWkYPpQrz5xiwqQBrXLvqVwAACAASURBVLRru8z75gBIlvrcAPg71d0E0SamkqQ9X1d9O+3zOWg4oqcGq9Ku7TLvm/U4kNsg+jtlgihJ2vPVaZ/PxSvbOGbBPYyf912OWXDPoJ4fr+wRPZcv+TK/mn84W/96NL+afzjLl3y5X95Xg19p13aZ981BlCx12yD6O2WCKEna85U0J2CZ6m0S9TInMV++5MtMXvEZDmQ9QwIOZD2TV3zGJFF9orRru8z75iBKlrpt6myWT7mcXzGGrRn8ijEsn3L5gPw7ZYIoSRoYps6u9F2Zv7HyPAD/6PZEvTW5LHNEz3EPLqQhXtn+veMVxj24sPD3Vj8rYWTNMq/txVuO4ZiXr2H8pps45uVrWLzlmMLfE6jbH/XOWX4ob930vzns5Zt466b/zTnLDx2QP+o5D6IkSXugsptc9rcyJzF/Q66H6Cy+ofD3Vj/acWTNjmkIoNDEpaxre8fBcTpaIdSWqVBTZw/qhHBHO/tRb6ANtGWCKEnSHqgeJ1EvaxLzZ2MMB7K+k/j+HNjvpVFh7r5i+2kXoLJ89xWFJzJlXNuDKWEZCAbTj3qFNjGNiFMiYm1EPB4R8zpZf2FErImIVRGxLCIm1ay7tLrf2oiY0d1jSpLUl8oaKMZJ1PvP0380l/YcsV2sPUfw9B/NLalEKkSdjaxZdsJST4NsQbn9qPtaYQliRAwFrgNOBSYBc2oTwKpvZuaUzJwGXAV8obrvJOBs4M3AKcD/iYih3TymJEl9osyBYmZNH8uVZ05hbGMDAYxtbHBOwIIcOfMCHvrjz203uMRDf/w5jpx5QfFvXkKfuLpVZyNrlpmw1NsgWzC4ftQrsonpUcDjmfkEQETcApwBPNKxQWb+umb71/H72SXPAG7JzJeBJyPi8erx2NUxpXq3eGVbKX14SrV6UaWJ0IutlT/0J15WV/0eVJyym2iV1eSyHh058wKoJoQHVh+FK6lPXN068bLtP28Y1CNrzp0xcbs+iNB/CUvZ984ylNmPuq8VmSCOBZ6uWW4Fjt5xo4j4JPApYATwzpp9f7zDvh2f7i6PKdWr0jukl8EvWCpQ2U20NMiV2CeuLnV8pnXyg2KZCUu93jsHy496RSaInYwHtq2G8PeBzOuA6yLiA8BngHN3sm9nTWJfc0yAiDgfOB/gkEMO6WaRpT5UQq1WPf5i5xcsFakeB4pRP6qzPnF7hDobWbOshMV758BW5CA1rcC4muUmYN1Otr8FmLWLfbt9zMy8PjObM7N5zJgxPSy61EsdtVovPg3k72u1Cu5bsm5jOzOHLGPZiIt5Yq8PsGzExcwcsmxw/2LnF6z+VWf9pQZTnxLtgeqsT5zqh/fOga3IBHE5MCEixkfECCqDziyp3SAiJtQsvgd4rPp6CXB2ROwVEeOBCcBPunNMaY+ws1qtAp27909YMPyfaBqygSEBTUM2sGD4P3Hu3j8p9H1L5Res/lPSDx9lcqAYFerEyyp94GoN4j5xqh/eOwe2wpqYZuarEXERsBQYCnw1Mx+OiCuAlsxcAlwUEe8CNgMvUGleSnW7RVQGn3kV+GRmbgHo7JhFnYO020qq1fr08FsZ9eor28VGxSt8evitwOWFvndp6mzQgVLVaXPewdKnRHugOusTp/rivXPgKrIPIpn5PeB7O8Quq3n9FzvZ9/PA57tzTGmPM7qpWsvSSbxAo9p/1aP4oOAXrP5jc16p79VZnzhJe75CE0SpbpVVq1VSYlo6v2D1j3q9viRJqiNF9kGU6tfU2XD6NTB6HBCV59OvKT6JsT+LiuT1JUnSoGcNolSUMmq1bG6pInl9SZI06EVmp9MIDirNzc3Z0tJSdjGkwa2EeR8lSZLUPRGxIjObd7WdNYiSeq9j+oOOPpcd0x+ASaIkSdIAYoIoqffqdPoDSZK0Z1m8so2FS9eybmM7Bzc2MHfGRKfb6CETREm95/QH/c8mvZIkbWfxyjYuvX0N7Zu3ANC2sZ1Lb18DYJLYA45iKqn3uprmwOkPitHRpPfFp4H8fZPe1YvKLpkkSaVZuHTttuSwQ/vmLSxcurakEg1MJoiSes/pD/rXzpr0SpJUp9ZtbO9RXJ0zQZTUe2XN+1ivbNIrSdJrHNzY0KO4OmcfREl9o4x5H+vV6KZq89JO4pIk1am5MyZu1wcRoGH4UObOmFhiqQYeaxAlaaCxSW//Wr0IvjgZ5jdWnu3rKUl7pFnTx3LlmVMY29hAAGMbG7jyzCkOUNND1iBK0kDTUVPrKKbFc45PSRpQZk0fa0LYSyaIkjQQ2aS3fzjHpySpztjEVJKkrjggkCSpzliDKKlPLF7ZxsKla1m3sZ2DGxuYO2OiTTw08DkgkCSpzliDKKnXFq9s49Lb19C2sZ0E2ja2c+nta1i8sq3sokm944BAkqQ6Y4IoqdcWLl273ZDSAO2bt7Bw6dqSSiT1Eef4lCTVGZuYSuq1dRvbexSXBhQHBJIk1RETREm9dnBjA22dJIMHNzZ0srW0e+znKklS8WxiKqnX5s6YSMPwodvFGoYPZe6MiSWVSION/VwlSeofu0wQI+KiiNinPwoj9bXFK9s4ZsE9jJ/3XY5ZcI9fJgsya/pYrjxzCmMbGwhgbGMDV545xdod9Rn7uUqS1D+608T0QGB5RDwIfBVYmplZbLE02JTRNGzxyjaW3fF/uJVbOHivDaz73f586Y6zgT83cSnArOlj/VxVGPu5SpLUP3ZZg5iZnwEmAF8BzgMei4j/FRFv2tW+EXFKRKyNiMcjYl4n6z8VEY9ExOqIuDsiDq3G3xERq2oemyJiVnXdjRHxZM26aT08Z/WzspqGrfru9VwR19M0ZANDApqGbOCKuJ5V372+0PeV1Pe66s9qP1dJkvpWt/ogVmsMf1V9vArsA9wWEVd1tU9EDAWuA04FJgFzImLSDputBJozcypwG3BV9f3uzcxpmTkNeCfwO+D7NfvN7Vifmau6cw4qT1lNw/7slW8wKl7ZLjYqXuHPXvlGoe8r9Yd6az5tP1dJkvrHLpuYRsTFwLnABuCfqCRnmyNiCPAY8Okudj0KeDwzn6ge5xbgDOCRjg0y896a7X8MfKiT45wF3JmZv9v16WhPVFbTsIOHPNejuDRQdNTKd/zw0lErDwzaZr4d5+UoppIkFas7fRD3B87MzF/WBjNza0SctpP9xgJP1yy3AkfvZPuPAXd2Ej8b+MIOsc9HxGXA3cC8zHx5J8dVycqaAmFTw4GMan+m83ih7ywVa2e18oM5YbKfqyRJxetOE9PvAc93LETE6yPiaIDMfHQn+0UnsU4Ht4mIDwHNwMId4gcBU4ClNeFLgSOAI4F9gUu6OOb5EdESES3r16/fSTFVtLKaho069QpeHTpyu9irQ0cy6tQrCn1fqWgO2CJJkorSnQTx74GXapZ/W43tSiswrma5CVi340YR8S7gr4CZndQEzgbuyMzNHYHMfCYrXgZuoNKU9TUy8/rMbM7M5jFjxnSjuCpKaVMgTJ3NsDP+DkaPAwJGj6ssT51d7PtKBXPAFkmSVJTuNDGN2mktqk1Lu7PfcmBCRIwH2qg0Ff3AdgeOmA58GTglM5/t5BhzqNQY1u5zUGY+ExEBzAIe6kZZVLLSmoZNnV1aQljG1B6qD3NnTNyuDyI4YIskSeob3Un0nqgOVNNRa/jnwBO72ikzX42Ii6g0Dx0KfDUzH46IK4CWzFxCpUnp3sC3KvkeT2XmTICIeCOVGsgf7nDomyJiDJUmrKuAC7txDlK/qsdBRNR/HLBFkiQVJXY1531EvAG4hsp0E0llYJi/7KLGb4/U3NycLS0tZRdDdeSYBfd0OjDP2MYG/mPeO0sokYpgLbEkSRooImJFZjbvartd1iBWE8Gz+6RUUp1wEJHBz1piSZI0GHVnHsSRVKageDOwbUjIzPxogeWSBrSypvZQ/6nXqSYkSdLg1p1RTP8ZOBCYQaU/YBPwmyILJQ10ZU3tof5jLbEkSRqMupMgHp6ZnwV+m5lfA95DZW5CSV0obWoP9RunmpAkSYNRd0Yx7ZiDcGNETAZ+BbyxsBJJg0RpU3uoXzjVhCRJGoy6kyBeHxH7AJ8BllCZluKzhZZKkvZwTjUhSZIGo50miBExBPh1Zr4A/Ag4rF9KJUkDgLXEkiRpsNlpH8TM3Apc1E9lkSRJkiSVqDuD1PwgIv5nRIyLiH07HoWXTJIkSZLUr7rTB7FjvsNP1sQSm5tKUt1ZvLLNfpeSJA1iu0wQM3N8fxREkrRnW7yybbuRW9s2tnPp7WsATBIlSRokdpkgRsQ5ncUz8+t9XxxJ0p5q4dK1203rAdC+eQsLl641QZQkaZDoThPTI2tejwROBB4ETBAlqY6s29jeo7gkSRp4utPE9P+tXY6I0cA/F1YiFca+Q5J64+DGBto6SQYPbmwooTSSJKkI3RnFdEe/Ayb0dUFUrI6+Q20b20l+33do8cq2sosmaYCYO2MiDcOHbhdrGD6UuTMmllQiSZLU17rTB/E7VEYthUpCOQlYVGSh1PfsOySptzruFbZEkCRp8OpOH8Sra16/CvwyM1sLKo8KYt8hSX1h1vSxJoSSJA1i3UkQnwKeycxNABHREBFvzMxfFFoy9Sn7Dmkws3+tJElS3+hOH8RvAVtrlrdUYxpASu87tHoRfHEyzG+sPK+2lbL6hv1rJUmS+k53EsRhmflKx0L19YjiiqQizJo+livPnMLYxgYCGNvYwJVnTumfWpbVi+A7F8OLTwNZef7OxSaJ6hM7618rSZKknulOE9P1ETEzM5cARMQZwIZii6UilNZ36O4rYPMOzVs3t1fiU2f3f3k0qNi/VpIkqe90J0G8ELgpIq6tLrcC5xRXJA06L3YxplFXcakH7F8rSZLUd3bZxDQzf56Zb6UyvcWbM/Ptmfl4dw4eEadExNqIeDwi5nWy/lMR8UhErI6IuyPi0Jp1WyJiVfWxpCY+PiIeiIjHIuLWiLC5655udFPP4lIPlN6/VpIkaRDZZYIYEf8rIhoz86XM/E1E7BMRn+vGfkOB64BTqSSXcyJi0g6brQSaM3MqcBtwVc269sycVn3MrIn/DfDFzJwAvAB8bFdlUclOvAyG71CbM7yhEpd6qdT+tZIkSYNMZObON4hYmZnTd4g9mJl/tIv93gbMz8wZ1eVLATLzyi62nw5cm5nHVJdfysy9d9gmgPXAgZn56o7v0ZXm5uZsaWnZ6XmqYKsXVfocvthaqTk88TL7H0qSJEn9JCJWZGbzrrbrTh/EoRGxV2a+XD1wA7BXN/YbCzxds9wKHL2T7T8G3FmzPDIiWoBXgQWZuRjYD9iYma/WHNNqgoFg6mwTQkmSJGkP150E8RvA3RFxQ3X5I8DXurFfdBLrtLoyIj4ENAPH14QPycx1EXEYcE9ErAF+3YNjng+cD3DIIYd0o7iSJEmSVN+6M0jNVcDngD+k0pfwLuDQne5U0QqMq1luAtbtuFFEvAv4K2BmRy1l9X3XVZ+fAO4DplOZXqMxIjoS206PWd3v+sxszszmMWPGdKO4kiRJklTfdpkgVv0K2Ar8KXAi8Gg39lkOTKiOOjoCOBtYUrtBtd/hl6kkh8/WxPeJiL2qr/cHjgEeyUqHyXuBs6qbngt8u5vnIEmSJEnaiS6bmEbE/0MlqZsDPAfcSmVQm3d058DVQWQuApYCQ4GvZubDEXEF0JKZS4CFwN7Atyrjz/BUdcTSPwS+HBFbqSSxCzLzkeqhLwFuqY6kuhL4Sk9PWpIkSZL0Wl2OYlpNzv4d+FjHvIcR8URmHtaP5esTjmIqSZIkqZ51dxTTnTUx/VMqTUvvjYh/jIgT6XzgGUmSJEnSINBlgpiZd2Tm+4EjqAwS8z+AAyLi7yPi5H4qn/rS6kXwxckwv7HyvHpR2SWSJEmStAfpziimv83MmzLzNCqjhq4C5hVeMvWt1YvgOxfDi08DWXn+zsUmiZIkSZK26e4opgBk5vOZ+eXMfGdRBVJB7r4CNrdvH9vcXolLkiRJEj1MEDWAvdjas7gkSZKkumOCWC9GN/UsLkmSJKnumCDWixMvg+EN28eGN1TikiRJkoQJYv2YOhtOvwZGjwOi8nz6NZW4JEmSJAHDyi6A+tHU2SaEkiRJkrpkDaIkSZIkCTBBlCRJkiRVmSBKkiRJkgATREmSJElSlQmiJEmSJAkwQZQkSZIkVZkgSpIkSZIAE0RJkiRJUpUJoiRJkiQJMEGUJEmSJFWZIEqSJEmSABNESZIkSVKVCaIkSZIkCTBBlCRJkiRVFZogRsQpEbE2Ih6PiHmdrP9URDwSEasj4u6IOLQanxYR90fEw9V176/Z58aIeDIiVlUf04o8B0mSJEmqF4UliBExFLgOOBWYBMyJiEk7bLYSaM7MqcBtwFXV+O+AczLzzcApwJciorFmv7mZOa36WFXUOUiSJElSPSmyBvEo4PHMfCIzXwFuAc6o3SAz783M31UXfww0VeM/y8zHqq/XAc8CYwosqyRJkiTVvSITxLHA0zXLrdVYVz4G3LljMCKOAkYAP68Jf77a9PSLEbFXXxRWkiRJkupdkQlidBLLTjeM+BDQDCzcIX4Q8M/ARzJzazV8KXAEcCSwL3BJF8c8PyJaIqJl/fr1u3cGkiRJklRHikwQW4FxNctNwLodN4qIdwF/BczMzJdr4n8AfBf4TGb+uCOemc9kxcvADVSasr5GZl6fmc2Z2TxmjK1TJUmSJGlXikwQlwMTImJ8RIwAzgaW1G4QEdOBL1NJDp+tiY8A7gC+npnf2mGfg6rPAcwCHirwHCRJkiSpbgwr6sCZ+WpEXAQsBYYCX83MhyPiCqAlM5dQaVK6N/CtSr7HU5k5E5gNHAfsFxHnVQ95XnXE0psiYgyVJqyrgAuLOgdJkiRJqieR2Wm3wEGlubk5W1payi6GJEmSJJUiIlZkZvOutiuyiakkSZIkaQAxQZQkSZIkASaIkiRJkqQqE0RJkiRJEmCCKEmSJEmqMkGUJEmSJAEmiJIkSZKkKhNESZIkSRJggihJkiRJqjJBlCRJkiQBJoiSJEmSpCoTREmSJEkSYIIoSZIkSaoyQZQkSZIkASaIkiRJkqQqE0RJkiRJEmCCKEmSJEmqMkGUJEmSJAEmiJIkSZKkKhNESZIkSRJggihJkiRJqjJBlCRJkiQBJoiSJEmSpKpCE8SIOCUi1kbE4xExr5P1n4qIRyJidUTcHRGH1qw7NyIeqz7OrYn/cUSsqR7zmoiIIs9BkiRJkupFYQliRAwFrgNOBSYBcyJi0g6brQSaM3MqcBtwVXXffYG/Bo4GjgL+OiL2qe7z98D5wITq45SizkGSJEmS6kmRNYhHAY9n5hOZ+QpwC3BG7QaZeW9m/q66+GOgqfp6BvCDzHw+M18AfgCcEhEHAX+QmfdnZgJfB2YVeA6SJEmSVDeKTBDHAk/XLLdWY135GHDnLvYdW33d3WNKkiRJkrppWIHH7qxvYHa6YcSHgGbg+F3s25Njnk+lKSqHHHLIrsoqSZIkSXWvyBrEVmBczXITsG7HjSLiXcBfATMz8+Vd7NvK75uhdnlMgMy8PjObM7N5zJgxu30SkiRJklQvikwQlwMTImJ8RIwAzgaW1G4QEdOBL1NJDp+tWbUUODki9qkOTnMysDQznwF+ExFvrY5eeg7w7QLPQZIkSZLqRmFNTDPz1Yi4iEqyNxT4amY+HBFXAC2ZuQRYCOwNfKs6W8VTmTkzM5+PiP+fSpIJcEVmPl99/QngRqCBSp/FO5EkSZIk9VpUBgMd3Jqbm7OlpaXsYkiSJElSKSJiRWY272q7IpuYSpIkSZIGEBNESZIkSRJggihJkiRJqjJBlCRJkiQBJoiSJEmSpCoTREmSJEkSYIIoSZIkSaoyQZQkSZIkASaIkiRJkqQqE0RJkiRJEmCCKEmSJEmqMkGUJEmSJAEwrOwC1KPFK9tYuHQt6za2c3BjA3NnTGTW9LFlF0uSJKkUmzdvprW1lU2bNpVdFGnAGzlyJE1NTQwfPny39jdB7GeLV7Zx6e1raN+8BYC2je1cevsaAJNESZJUl1pbW3n961/PG9/4RiKi7OJIA1Zm8txzz9Ha2sr48eN36xg2Me1nC5eu3ZYcdmjfvIWFS9eWVCJJkqRybdq0if3228/kUOqliGC//fbrVW28CWI/W7exvUdxSZKkemByKPWN3v5fMkHsZwc3NvQoLkmSpOJ99KMf5Q1veAOTJ0/ucpvM5OKLL+bwww9n6tSpPPjgg9vWfe1rX2PChAlMmDCBr33ta7t8v/vuu4/TTjutT8q+O2688UbWrVtX2vv3tbvuuouJEydy+OGHs2DBgk63+cIXvsCkSZOYOnUqJ554Ir/85S+3rRs6dCjTpk1j2rRpzJw5c5fv94tf/GKn10rRFi9ezCOPPFLIsU0Q+9ncGRNpGD50u1jD8KHMnTGxpBJJkiQNLItXtnHMgnsYP++7HLPgHhavbOv1Mc877zzuuuuunW5z55138thjj/HYY49x/fXX84lPfAKA559/nssvv5wHHniAn/zkJ1x++eW88MILu1WOV199dbf266lSE8TVi+CLk2F+Y+V59aJeHW7Lli188pOf5M477+SRRx7h5ptv7jR5mj59Oi0tLaxevZqzzjqLT3/609vWNTQ0sGrVKlatWsWSJUt2uyz99e9ngjiIzJo+livPnMLYxgYCGNvYwJVnTnGAGkmSpG7oGPCvbWM7ye8H/Ottknjcccex77777nSbb3/725xzzjlEBG9961vZuHEjzzzzDEuXLuWkk05i3333ZZ999uGkk07qNNm86667OOKIIzj22GO5/fbbt8Xnz5/P+eefz8knn8w555zDpk2b+MhHPsKUKVOYPn069957L1BJ6s444wxOOeUUJk6cyOWXX77tGF/4wheYPHkykydP5ktf+hLw2lquq6++mvnz53PbbbfR0tLCBz/4QaZNm0Z7ez92dVq9CL5zMbz4NJCV5+9c3Ksk8Sc/+QmHH344hx12GCNGjODss8/m29/+9mu2e8c73sGoUaMAeOtb30pra2uP3mfFihW85S1v4W1vexvXXXfdtviNN97I+973Pk4//XROPvlkMpO5c+cyefJkpkyZwq233gpUao2PO+443vve9zJp0iQuvPBCtm7dCsDNN9/MlClTmDx5Mpdccsm2Y++9997bXt92222cd955/Od//idLlixh7ty5TJs2jZ///Oc9Oo9dcRTTEsyaPtaEUJIkaTfsbMC/or9ftbW1MW7cuG3LTU1NtLW1dRmvtWnTJj7+8Y9zzz33cPjhh/P+979/u/UrVqxg2bJlNDQ08Ld/+7cArFmzhp/+9KecfPLJ/OxnPwMqydBDDz3EqFGjOPLII3nPe95DRHDDDTfwwAMPkJkcffTRHH/88eyzzz6dnsdZZ53Ftddey9VXX01zc3OffDbddvcVsHmHhHRzeyU+dfZuHbKzz/+BBx7Y6T5f+cpXOPXUU7ctb9q0iebmZoYNG8a8efOYNWvWa/b5yEc+wt/93d9x/PHHM3fu3O3W3X///axevZp9992Xf/mXf2HVqlX813/9Fxs2bODII4/kuOOOAyr/fo888giHHnoop5xyCrfffjtvf/vbueSSS1ixYgX77LMPJ598MosXL+60DABvf/vbmTlzJqeddhpnnXVWtz+n7rIGUZIkSQNGmQP+ZeZrYhHRZbzWT3/6U8aPH8+ECROICD70oQ9tt37mzJk0NFTGpFi2bBkf/vCHATjiiCM49NBDtyWIJ510Evvttx8NDQ2ceeaZLFu2jGXLlvHe976X173udey9996ceeaZ/Pu//3ufnHOfe7GLWruu4t3Qnc+/1je+8Q1aWlq2S/KeeuopWlpa+OY3v8lf/uVfvqZW7sUXX2Tjxo0cf/zxANv+fTp01CBD5d9vzpw5DB06lAMOOIDjjz+e5cuXA3DUUUdx2GGHMXToUObMmcOyZctYvnw5J5xwAmPGjGHYsGF88IMf5Ec/+tHufRh9wASxDH3c7lqSJKlelDngX1NTE08//fS25dbWVg4++OAu4zvaWdLyute9btvrzhKero7RVYIKMGzYsG1NGIFeTX3QZ0Y39SzeDd39/AH+7d/+jc9//vMsWbKEvfbaa1u8Y/vDDjuME044gZUrV263X2b2+7/fjtv3179foQliRJwSEWsj4vGImNfJ+uMi4sGIeDUizqqJvyMiVtU8NkXErOq6GyPiyZp104o8hz5XQLtrSZKkelHmgH8zZ87k61//OpnJj3/8Y0aPHs1BBx3EjBkz+P73v88LL7zACy+8wPe//31mzJix3b5HHHEETz755LaaqZtvvrnL9znuuOO46aabAPjZz37GU089xcSJlfP7wQ9+wPPPP097ezuLFy/mmGOO4bjjjmPx4sX87ne/47e//S133HEHf/Inf8IBBxzAs88+y3PPPcfLL7/Mv/7rv257j9e//vX85je/6euPaNdOvAyG75DMD2+oxHfTkUceyWOPPcaTTz7JK6+8wi233NLpSKQrV67kggsuYMmSJbzhDW/YFn/hhRd4+eWXAdiwYQP/8R//waRJk7bbt7GxkdGjR7Ns2TKAbf8+nTnuuOO49dZb2bJlC+vXr+dHP/oRRx11FFBpYvrkk0+ydetWbr31Vo499liOPvpofvjDH7Jhwwa2bNnCzTffvK2m8oADDuDRRx9l69at3HHHHdveo8h/v8L6IEbEUOA64CSgFVgeEUsys3a4naeA84D/WbtvZt4LTKseZ1/gceD7NZvMzczbiip7oQpody1JklQvOvoZLly6lnUb2zm4sYG5Myb2uv/hnDlzuO+++9iwYQNNTU1cfvnlfOxjH+Mf/uEfALjwwgt597vfzfe+9z0OP/xwRo0axQ033ADAvvvuy2c/+1mOPPJIAC677LLXDHgzcuRIrr/+et7znvew//77c+yxx/LQQw91ZC8M9gAACIBJREFUWpY///M/58ILL2TKlCkMGzaMG2+8cVtt17HHHsuHP/xhHn/8cT7wgQ9s60N43nnnbUtC/uzP/ozp06dvK8vRRx/N+PHjOeKII7a9x3nnnceFF15IQ0MD999//7bmrYXr+L579xWVZqWjmyrJYS++Bw8bNoxrr72WGTNmsGXLFj760Y/y5je/Gaicf3NzMzNnzmTu3Lm89NJLvO997wPgkEMOYcmSJTz66KNccMEFDBkyhK1btzJv3rzXJIgAN9xwAx/96EcZNWrUa34AqPXe976X+++/n7e85S1EBFdddRUHHnggP/3pT3nb297GvHnzWLNmzbYBa4YMGcKVV17JO97xDjKTd7/73ZxxxhkALFiwgNNOO41x48YxefJkXnrpJQDOPvtsPv7xj3PNNddw22238aY3vWm3P78dxc6qNHt14Ii3AfMzc0Z1+VKAzLyyk21vBP61s6QvIs4Hjs/MD+5q2640NzdnS0vL7pxG35vfCHT2mQfM39jfpZEkSSrdo48+yh/+4R+WXYw93o033khLSwvXXntt2UXRbrjvvvu4+uqrt6vJLUpn/6ciYkVm7nJUoiKbmI4Fnq5Zbq3GeupsYMc6+M9HxOqI+GJE7NXZTnusAtpdS5IkSVJfKDJB7KwXZ4+qKyPiIGAKsLQmfClwBHAksC9wSSe7EhHnR0RLRLSsX7++J29brALaXUuSJGnwO++886w9HMBOOOGEfqk97K0iE8RWYFzNchOwrofHmA3ckZmbOwKZ+UxWvAzcABzV2Y6ZeX1mNmdm85gxY3r4tgWaOhtOvwZGjwOi8nz6NfY/lCRJklS6wgapAZYDEyJiPNBGpanoB3p4jDlUagy3iYiDMvOZqIz5OgvovHfvnmzqbBNCSZKkGruaRkBS9/R2jJnCahAz81XgIirNQx8FFmXmwxFxRUTMBIiIIyOiFXgf8OWIeLhj/4h4I5UayB/ucOibImINsAbYH/hcUecgSZKk4o0cOZLnnnuu119spXqXmTz33HOMHDlyt49R2Cime5I9ahRTSZIkbWfz5s20trbuGRO5SwPcyJEjaWpqYvjw4dvFuzuKaZFNTCVJkqRdGj58OOPHjy+7GJIodpAaSZIkSdIAYoIoSZIkSQJMECVJkiRJVXUxSE1ErAd+WXY5OrE/sKHsQmjQ8vpSkby+VCSvLxXNa0xF2lOvr0Mzc5cTxNdFgrinioiW7owkJO0Ory8VyetLRfL6UtG8xlSkgX592cRUkiRJkgSYIEqSJEmSqkwQy3V92QXQoOb1pSJ5falIXl8qmteYijSgry/7IEqSJEmSAGsQJUmSJElVJogliIhTImJtRDweEfPKLo8Gn4j4RUSsiYhVEdFSdnk0sEXEVyPi2Yh4qCa2b0T8ICIeqz7vU2YZNXB1cX3Nj4i26j1sVUS8u8wyauCKiHERcW9EPBoRD0fEX1Tj3sPUazu5vgb0Pcwmpv0sIoYCPwNOAlqB5cCczHyk1IJpUImIXwDNmbknzsGjASYijgNeAr6emZOrsauA5zNzQfWHrn0y85Iyy6mBqYvraz7wUmZeXWbZNPBFxEHAQZn5YES8HlgBzALOw3uYemkn19dsBvA9zBrE/ncU8HhmPpGZrwC3AGeUXCZJ6lJm/gh4fofwGcDXqq+/RuUPotRjXVxfUp/IzGcy88Hq698AjwJj8R6mPrCT62tAM0Hsf2OBp2uWWxkEF5L2OAl8PyJWRMT5ZRdGg9IBmfkMVP5AAm8ouTwafC6KiNXVJqg2/1OvRcQbgenAA3gPUx/b4fqCAXwPM0Hsf9FJzHa+6mvHZOYfAacCn6w24ZKkgeLvgTcB04BngL8ttzga6CJib+BfgL/MzF+XXR4NLp1cXwP6HmaC2P9agXE1y03AupLKokEqM9dVn58F7qDStFnqS/9d7XvR0Qfj2ZLLo0EkM/87M7dk5lbgH/Eepl6IiOFUvrzflJm3V8Pew9QnOru+Bvo9zASx/y0HJkTE+IgYAZwNLCm5TBpEIuJ11Y7SRMTrgJOBh3a+l9RjS4Bzq6/PBb5dYlk0yHR8ca96L97DtJsiIoCvAI9m5hdqVnkPU691dX0N9HuYo5iWoDrU7ZeAocBXM/PzJRdJg0hEHEal1hBgGPBNrzH1RkTcDJwA7A/8N/DXwGJgEXAI8BTwvsx0oBH1WBfX1wlUmmYl8Avggo7+YlJPRMSxwL8Da4Ct1fD/R6WfmPcw9cpOrq85DOB7mAmiJEmSJAmwiakkSZIkqcoEUZIkSZIEmCBKkiRJkqpMECVJkiRJgAmiJEmSJKnKBFGSpG6KiC0RsarmMa8Pj/3GiBhQc2VJkgafYWUXQJKkAaQ9M6eVXQhJkopiDaIkSb0UEb+IiL+JiJ9UH4dX44dGxN0Rsbr6fEg1fkBE3BER/1V9vL16qKER8Y8R8XBEfD8iGko7KUlSXTJBlCSp+xp2aGL6/pp1v87Mo4BrgS9VY9cCX8/MqcBNwDXV+DXADzPzLcAfAQ9X4xOA6zLzzcBG4E8LPh9JkrYTmVl2GSRJGhAi4qXM3LuT+C+Ad2bmExExHPhVZu4XERuAgzJzczX+TGbuHxHrgabMfLnmGG8EfpCZE6rLlwDDM/NzxZ+ZJEkV1iBKktQ3sovXXW3TmZdrXm/BsQIkSf3MBFGSpL7x/prn+6uv/xM4u/r6g8Cy6uu7gU8ARMTQiPiD/iqkJEk74y+TkiR1X0NErKpZviszO6a62CsiHqDy4+ucauxi4KsRMRdYD3ykGv8L4PqI+BiVmsJPAM8UXnpJknbBPoiSJPVStQ9ic2ZuKLsskiT1hk1MJUmSJEmANYiSJEmSpCprECVJkiRJgAmiJEmSJKnKBFGSJEmSBJggSpIkSZKqTBAlSZIkSYAJoiRJkiSp6v8CcUIHnlCNRSAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb2cec96e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). How should we modify p, if at all, if we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answer:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
